# Validation {#validation}

```{r setup4, include=FALSE, echo = FALSE, message = FALSE, warning=FALSE}
knitr::opts_chunk$set(message = FALSE)
rm(list = ls())  # delete everything that is in R's memory
options(scipen=1000) # removal of scientific notation

path <- "C:/Users/PouwelsXGLV/Documents/Maastricht/Eribulin CEA/Model final_Gamma/" #set location of model, to adapt!
setwd(paste(path,"report", sep = "")) # set working directory to File Source

#load package
library(rms)
library(survival)
library(muhaz)
library(survminer)
library(ggplot2)
library(data.table)
library(printr)
library(dplyr)
library(flexsurv)
library(plyr)
library(tidyverse)
library(scales)
library(matrixStats)
library(reshape2)

# load functions
source(paste(path,"R/01_fun_analyses.R", sep = ""))
source(paste(path,"R/03_fun_dowsa.R", sep = ""))
source(paste(path,"R/03_fun_scenarios.R", sep = ""))
source(paste(path,"R/03_fun_psa_inputs.R", sep = ""))
source(paste(path,"R/03_fun_threshold.R", sep = ""))

# load model inputs
load(paste(path,"data/01_model_inputs.R", sep = ""))
```

This Section presents the validation efforts undertaken during the development of the model to ensure its validity. These validation efforts are reported according to the AdVIShe reporting tool[@Vemer2016].

## A. Validation of the conceptual model

### A1. Face validity
The conceptual model has been developed and discussed with clinical experts (MdB, VCTH). Both of these experts have extensive experience in treating advanced breast cancer patients.  
The clinical experts agreed with modelling advanced disease as in the present study.

### A2. Cross validity testing
The conceptual model has the same structure as previous models investigating chemotherapeutic treatment for advanced breast cancer patients [@Pouwels2017].

## B. Input data validation
The source of all inputs data are described in Tables 1 and 2 and in the methods section. All these input data have been obtained from Dutch sources, except for the disutility value associated with adverse event.  
Additionally, we have compared the annual transition probabilities estimated by the PFS, OS and TTD survival curves with the age-adjusted Dutch mortality probability [REF CBS]. None of the annual estimated survival probabilities calculated by the survival models were smaller than the age-adjusted Dutch mortality probability.  

### B1. Face validity
A clinical expert (MdB) has been asked whether treatment dependent or independent resource use inputs should inform the model. MdB argued that treatment indepdent resource use should be used. Additionally, the assumption of no productivity losses has been informed by clinical expert opinion (MdB).  

### B2. Model fit testing
The statistical fit statistics (Akaike Information Criterion) of the survival curves have been computed and used for model selection.

## C. Validation of the computerised model
The only difference between the computerised model and the conceptual model is that the transition from the progressed disease to the dead health state is not explicitly modelled. This is a limitation is caused by the use of a partitioned survival model.

### C1. External review
The model has been externally reviewed by an independent modeler. The reviewer checked whether the different functions used in the model were working and reproduced the results. Additionally, the external reviewer performed some of the tests described in C2.

### C2. Extreme value testing
The model has been run with 1) discounting rates of 0% per year, 2) health state utility values of 1 (and no disutility for adverse events), 3) without any costs, 4) considering the same PFS, OS and TTD curves for each arms. In each case the results were as expected: 1) same discounted and undiscounted total outcomes, 2) same total life years and quality-adjusted life years, 3) total costs of each arm were equal to 0, 4) the outcomes in both groups were equal. The tests number 1) and 2), and 3) were performed by XP, and the tests number 1), 2) and 4) were performed by an external reviewer.

### C3. Testing of traces
The traces were computed and discussed between XP, BR, and MdB. The traces are available [here](#fig:fig-traces).

### C4. Unit testing
The functions estimating the survival in each group have been visually compared to the Kaplan-Meier curves.  

## D. Operational validation

### D1. Face validity testing
Aggregated and disaggregated total costs, life years, and quality-adjusted life years have been presented to all the co-authors. As expected, the acquisition costs of eribulin are the main cost drivers. A clinical expert (MdB) was also asked whether the incremental cost effectiveness ratio was as expected and she responded she expected eribulin not to be cost effective. The current results confirmed her belief.  

### D2. Cross validation testing
Our results confirm Lopes et al.'s conclusion that eribulin is not cost effective as third (or later) treatment line for ABC patients [@Lopes2013]. In the study of Lopes et al., eribulin led to 0.119 incremental QALY, which is slightly higher than in the present study (0.07). This difference may be explained by the fact that the present study is based on routinely collected clinical data while Lopes' et al. was informed by the EMBRACE trial.  

### D3. Validation against outcomes using alternative input data
Multiple scenario analyses have been performed using alternative input data (utility values, relative effectiveness and alternative survival models). The results of these scenarios were not dramatically different than our base-case analysis. The results of these analyses are reported in Table 4 of the paper.

### D4. Validation against empirical data
The median survival obtained from the parametric survival models has been compared to the median survival obtained from the Kaplan-Meier curves for overall survival and progression-free survival. As can be seen in Table \@ref(tab:valid), the modelled OS of both groups is overestimated by approximately 6 months while the PFS is a bit underestimated.  

```{r, echo = FALSE}
#OS
#median modelled survival eribulin
s.OS.trt <- params_det$s.OS.trt
s.OS.trt <- s.OS.trt[which(as.numeric(as.character(s.OS.trt[,params_det$s.mod.OS]))>0.5),]
s.OS.trt.median <- round(s.OS.trt[which.min(abs(0.5 - as.numeric(as.character(s.OS.trt[,params_det$s.mod.OS])))), "Time"]*365.25/30.4375,1)

#median modelled survival non-eribulin
s.OS.comp <- params_det$s.OS.comp
s.OS.comp <- s.OS.comp[which(as.numeric(as.character(s.OS.comp[,params_det$s.mod.OS]))>0.5),]
s.OS.comp.median <- round(s.OS.comp[which.min(abs(0.5 - as.numeric(as.character(s.OS.comp[,params_det$s.mod.OS])))), "Time"]*365.25/30.4375,1)

#PFS
#median modelled survival eribulin
s.PFS.trt <- params_det$s.PFS.trt
s.PFS.trt <- s.PFS.trt[which(as.numeric(as.character(s.PFS.trt[,params_det$s.mod.PFS]))>0.5),]
s.PFS.trt.median <- round(s.PFS.trt[which.min(abs(0.5 - as.numeric(as.character(s.PFS.trt[,params_det$s.mod.PFS])))), "Time"]*365.25/30.4375,1)

#median modelled survival non-eribulin
s.PFS.comp <- params_det$s.PFS.comp
s.PFS.comp <- s.PFS.comp[which(as.numeric(as.character(s.PFS.comp[,params_det$s.mod.PFS]))>0.5),]
s.PFS.comp.median <- round(s.PFS.comp[which.min(abs(0.5 - as.numeric(as.character(s.PFS.comp[,params_det$s.mod.PFS])))), "Time"]*365.25/30.4375,1)

```


Table: (\#tab:valid) Comparison of the modelled and observed OS and PFS.

| Outcome | Group | Observed median survival in months [@Pouwels2019] |Modelled median survival|
|:--------|:------|:-------------------------|:-------------------------|
|Overall survival| Eribulin|5.9 months| `r s.OS.trt.median` months|
|Overall survival| Non-eribulin|5.2 months| `r s.OS.comp.median` months|
|Progression-free survival| Eribulin|3.5 months| `r s.PFS.trt.median` months|
|Progression-free survival| Non-eribulin|3.2 months| `r s.PFS.comp.median` months|


## E. Other validation technique
None
